Optimizing Large Language Model Serving: The Critical Shift from Throughput to Goodput

The rapid ascent of Large Language Models (LLMs) has revolutionized artificial intelligence, pushing the boundaries of what machines can achieve in language understanding and generation. However, deploying these sophisticated models in production environments at scale presents a unique set of challenges, particularly concerning performance measurement. On July 20, 2026, a significant insight emerged from Graziano Casto of Akamas, highlighting a fundamental flaw in the prevailing metric for LLM serving performance: raw throughput. Published through the Cloud Native Computing Foundation (CNCF), this research, leveraging projects like Kubernetes and Prometheus, unequivocally argues that focusing solely on how many requests an LLM system can process per second, while seemingly logical and financially appealing, can lead to a deceptive sense of efficiency, ultimately degrading user experience. The critical distinction lies in moving from "throughput" to "goodput" – a metric that accounts not just for quantity, but also for quality, ensuring that served requests meet crucial latency targets.

The conventional wisdom in benchmarking LLM serving setups has long gravitated towards throughput, primarily due to its ease of measurement, straightforward comparability, and direct correlation with cost-efficiency (dollars per request). Engineers often chase this metric, meticulously tuning parameters like batching strategies, concurrency levels, and GPU utilization, all in pursuit of higher requests per second. While this approach might superficially indicate a busy and productive system, Casto’s research demonstrates that a rising throughput number can mask a silently deteriorating service quality. This paradox was vividly illustrated in experiments tuning vLLM on a single GPU, revealing a substantial gap between raw processing power and actual user satisfaction.

The Nuance of LLM Performance: Beyond Raw Throughput

To illustrate this complex relationship, Casto employs a compelling analogy: an LLM endpoint as a bustling restaurant kitchen. In this scenario, throughput is akin to the number of plates leaving the pass every hour. A high plate count signifies a busy kitchen, but offers no guarantee about the dining experience. Did the food arrive hot? Was it on time? Was it even delivered to the correct table? A kitchen could ostensibly double its plate output, yet simultaneously generate a full room of unhappy diners served cold, late, or incorrect meals.

The metric that truly captures the essence of a positive user experience is "goodput." This definition, refreshingly plain, quantifies the number of completed requests per second that also successfully meet predefined latency targets. For LLM serving, these targets typically include Time To First Token (TTFT), which measures the delay before any output begins to appear on screen, and Time Per Output Token (TPOT), which dictates the streaming pace of the subsequent response. A request that finishes but fails to meet either of these critical targets is analogous to a plate that left the kitchen cold; it contributes to raw throughput but registers as a failure in terms of goodput. This distinction is paramount, as user perception of responsiveness is heavily influenced by these temporal factors. A user waiting for a chatbot response prioritizes an immediate initial reply (low TTFT) and a smooth, continuous stream of information (low TPOT). Any significant delay or stutter can lead to frustration and abandonment, irrespective of the system’s underlying processing capacity.

The Experimental Framework: A Deep Dive into the Akamas Setup

To rigorously investigate this performance dichotomy, Casto established a meticulously controlled experimental environment, utilizing components widely accessible within modern cloud-native infrastructures. The chosen Large Language Model for the study was Qwen2.5-7B, a moderately sized model, served by vLLM, an open-source library renowned for its high-throughput inference capabilities. The hardware foundation comprised a single NVIDIA A10G GPU, equipped with 24 GB of memory, hosted within an Amazon Elastic Kubernetes Service (EKS) cluster. This choice of a single A10G, while not representative of the most powerful production-grade LLM services that might leverage H100 clusters, was deliberate. It allowed for frequent, cost-effective studies, ensuring that the insights derived focused on the fundamental trade-off shape between throughput and latency, a principle that remains consistent regardless of GPU scale.

Load generation, a critical aspect of any performance benchmark, was managed by GuideLLM, a specialized tool developed for driving synthetic traffic to LLM endpoints. For real-time system observation and data collection, the familiar combination of Prometheus and Grafana was deployed. GPU-level metrics, providing crucial insights into hardware utilization, were captured via NVIDIA’s DCGM exporter, feeding directly into the Prometheus monitoring stack. The entire experimental setup, including Kubernetes manifests, has been openly shared on GitHub, empowering other researchers and practitioners to reproduce the runs or adapt the benchmark to their own models and specific traffic patterns, moving beyond synthetic loads to real-world scenarios. This commitment to transparency and reproducibility is a hallmark of robust scientific inquiry in the cloud-native domain.

Precision Tuning: The VLLM Parameters Under Scrutiny

The core of Casto’s investigation revolved around systematically varying three specific vLLM configuration settings, chosen for their direct influence on the throughput-latency trade-off:

1. gpu_memory_utilization: This parameter dictates the proportion of GPU memory allocated for the KV cache, which stores key-value pairs of attention states, critical for accelerating subsequent token generation. A higher utilization can mean more active sequences, potentially boosting throughput but also risking memory contention.
2. max_num_batched_tokens: This setting controls the maximum number of tokens processed in a single batch step. Larger batches can improve GPU efficiency and throughput by keeping the compute units saturated, but they can also introduce longer latencies for individual requests waiting for a batch to fill.
3. max_num_seqs: This parameter sets the maximum number of concurrent sequences (individual requests) that vLLM can handle simultaneously. Increasing concurrency can enhance throughput, but at the cost of potentially longer queues and increased latency for each sequence as resources are shared.

These three parameters were intentionally selected because they represent knobs that operators can safely adjust in a production environment without altering the underlying model, hardware, or cluster configuration. By holding all other variables constant, the study ensured that the observed performance changes were directly attributable to these specific vLLM settings, simplifying the interpretation of results and enhancing their practical applicability.

Driving the Load: Methodology for Realistic Benchmarking

The method by which traffic is sent to the LLM service is as crucial as the tuning parameters themselves. Casto employed GuideLLM in two distinct modes to gain comprehensive insights:

1. Rate Sweep: This approach gradually increases the request rate over time, allowing the system to stabilize at various load levels. This method is instrumental in observing how latencies behave while the server is still capable of keeping pace with incoming requests, providing a realistic view of performance under normal, non-saturated conditions.
2. Throughput Run: In contrast, this method pushes the server to its absolute limits by firing requests as rapidly as the system can accept them. This aggressively saturates the server, revealing its maximum raw throughput capacity, albeit often at the expense of significantly elevated latencies.

The strategic use of both methods provided a nuanced understanding of the system’s behavior across a spectrum of operational demands, offering insights into both ideal-state performance and peak-capacity limits. The contrast between these two load-driving techniques formed a critical part of the study’s key lessons.

Workload Diversity: Tailoring Benchmarks to Application Needs

Recognizing that different LLM applications have varying performance requirements, the study evaluated the setup against three distinct types of traffic, each designed to highlight specific latency sensitivities:

1. Chatbot Workload: Characterized by relatively short prompts (~485 tokens) and commensurately short answers (~121 tokens). For this workload, a tight TTFT is paramount, ensuring a near-instantaneous initial response, coupled with a medium TPOT for a fluid conversational flow.
2. Reasoning Workload: Involves short prompts (~400 tokens) but demands very long answers, often on the order of 4,000 tokens. Here, TPOT becomes the dominant latency concern, as the user waits for a lengthy, detailed response to stream out, while TTFT can be comparatively looser.
3. Agentic Workload: Consists of many successive short calls (~512 token prompts, ~128 token outputs) chained together to complete a complex task. The cumulative effect of per-call TTFT is critical here, as delays in each step can significantly prolong the overall task completion time.

This diversified workload approach ensured that the findings were relevant across a broad spectrum of real-world LLM applications, underscoring that a "one-size-fits-all" optimization strategy is insufficient.

The Unveiling of Critical Findings: Throughput vs. Latency Trade-offs

The chatbot study provided the clearest and most compelling evidence of the throughput-goodput divergence. The primary objective was to maximize combined token throughput (prefill_token_throughput + decode_token_throughput) under a stringent constraint: the average TTFT had to remain at or below 1.5 seconds. To mitigate the inherent jitter in live metrics, a windowing rule was applied, scoring each configuration based on a stretch of eight consecutive data points where prefill throughput had stabilized, rather than relying on potentially anomalous single samples.

Analyzing two exemplary configurations, Exp 1 and Exp 20, revealed striking differences. Both configurations successfully adhered to the imposed TTFT constraint, landing at approximately 1.4 seconds. This wasn’t coincidental; it was the constraint functioning precisely as intended, forcing the system to maintain a user-acceptable first-token wait. However, in terms of combined throughput, Exp 20 emerged as the clear winner, boasting approximately 50 percent more combined throughput than Exp 1.

This significant gain in throughput, however, came at a substantial, unconstrained cost: the Time Per Output Token (TPOT). Exp 1 achieved an impressively low p95 TPOT of about 50 milliseconds, ensuring a virtually seamless streaming experience. In stark contrast, Exp 20’s p95 TPOT soared to approximately 494 milliseconds, nearly ten times higher. While both configurations fulfilled the explicit TTFT promise, the near half-second gap between successive tokens in Exp 20 would translate into a visibly stuttering, frustrating experience for a user reading a chat response as it streamed in. This illustrates the core problem: optimizing for a single metric (TTFT in this case, to enable higher throughput) can inadvertently compromise another crucial aspect of user experience (TPOT).

The underlying mechanism for this trade-off is rooted in the distinct stages of an LLM request. TTFT is primarily influenced by the prefill stage, where the model processes the initial prompt. By bounding TTFT, the optimization ensures that the initial processing remains swift. Subsequent tokens are generated during the decode stage, where vLLM concurrently advances all active requests, generating one token for each per step. To achieve higher combined throughput, more work is packed into each decode step, meaning a fuller step takes longer to execute. Consequently, every request experiences a slightly longer wait for its next token. Metaphorically, if the kitchen is constrained to seat diners quickly and serve the first course promptly, the increased load manifests as longer waits between subsequent courses.

Broader Implications: Reasoning and Agentic Workloads Echo the Warning

The reasoning and agentic workload studies, though conducted under different constraints (maximizing throughput with no TTFT constraint, using a throughput run), reinforced the same fundamental lesson. In the agentic study, two top-performing configurations exhibited nearly identical requests per second (6.20 vs. 6.11), yet their p95 TPOT diverged by approximately 220 milliseconds (682 ms vs. 902 ms). For a multi-step agentic task, this seemingly small difference in inter-token latency can accumulate into a significantly slower overall task completion time, directly impacting the agent’s utility and perceived responsiveness.

Similarly, in the reasoning study, where answers could extend to 4,000 tokens, the configuration yielding the highest raw throughput proved suboptimal when latency became a concern. Imposing a tighter inter-token target (TPOT) during the optimization process led to an entirely different winning configuration. Across all three workload types and varied experimental objectives, the consistent conclusion emerged: a raw throughput number, devoid of associated latency metrics, is insufficient for informed configuration decisions. It fails to convey the actual user experience, which is the ultimate arbiter of service quality.

Strategic Imperatives for Production AI: Optimizing for User Experience

From this comprehensive research, four critical lessons emerge for anyone responsible for deploying and managing LLM inference in production:

1. Goodput, Not Just Throughput, is the True North: The most significant takeaway is that a throughput number without a clearly defined Service Level Objective (SLO) for latency is largely a marketing figure, not a robust engineering metric. The honest target for optimization must be "goodput," encompassing requests per second that demonstrably meet specified TTFT and TPOT targets. Failing to account for latency means counting "cold plates" – requests processed but delivering a poor user experience.
2. Automated, Data-Driven Search is Essential: The optimal configuration for goodput is rarely intuitively obvious. As demonstrated in the reasoning runs, seemingly minor adjustments can lead to vastly different performance outcomes, with "small cliffs" in the performance landscape. Manual tuning often settles for the first "good enough" number rather than discovering the truly optimal one. A reliable path to peak goodput involves automated, data-driven search mechanisms that systematically explore a wide array of configurations against real, user-centric targets, rather than relying on a few educated guesses. Akamas’s expertise in AI-driven optimization aligns perfectly with this imperative, emphasizing intelligent exploration of parameter spaces.
3. Optimization is a Continuous Process: The landscape of LLM deployment is inherently dynamic. Factors such as incoming traffic patterns, the nature of prompts, and even updates to the models themselves evolve constantly. Consequently, what constitutes the "best" configuration is not static; it shifts with these changing conditions. This necessitates an ongoing, iterative optimization process, where the search for goodput is not a one-time event but a continuous endeavor.
4. Full-Stack Tuning Yields Maximum Gains: While Casto’s study focused on three vLLM-specific parameters, the trade-offs between throughput and latency extend across the entire technology stack. GPU settings, runtime environments, and scheduling algorithms all play a role, and their interactions are complex. The most substantial performance improvements often stem from a holistic, full-stack tuning approach, rather than isolated optimizations at individual layers. Kubernetes, with its robust orchestration capabilities, provides an ideal platform for managing and optimizing these interconnected layers.

The practical implication for MLOps engineers and product managers is clear and actionable: prioritize the establishment of explicit TTFT and TPOT targets first. Only then should efforts be directed towards identifying the configuration that yields the highest throughput while consistently meeting those latency objectives. This strategic shift may sometimes result in a numerically lower "requests-per-second" figure, but it invariably translates into a demonstrably superior and more reliable service – a trade-off that end-users will consistently value and appreciate.

The Road Ahead: Continuous Optimization and Full-Stack Tuning

The findings presented by Graziano Casto and Akamas, championed by the CNCF, mark a pivotal moment in the maturity of LLM operations. They underscore the necessity for the AI community to adopt more sophisticated, user-centric metrics for evaluating performance. The era of simply measuring raw computational output is waning; the future belongs to systems optimized for genuine user satisfaction. The tools are already available within the cloud-native ecosystem. Kubernetes provides the robust orchestration layer, Prometheus and Grafana offer unparalleled observability, and specialized benchmarking tools like GuideLLM facilitate rigorous testing. The integration of NVIDIA’s DCGM exporter ensures that GPU-level insights are not overlooked, allowing for fine-grained control and optimization at the hardware level.

For organizations seeking to bridge the gap between impressive raw benchmarks and actual user delight, the path is clear. The full setup demonstrated in this research is openly available on GitHub at github.com/graz-dev/vllm-benchmark. By cloning the repository, directing GuideLLM towards their specific models and traffic profiles, and diligently setting their own Service Level Objectives, practitioners can embark on a journey to uncover the "real headroom" – the hidden potential for enhanced goodput that truly defines a superior LLM service. This continuous pursuit of goodput, rather than mere throughput, will be the differentiator for successful, scalable, and user-delighting AI applications in the years to come.